General Disclaimer 


One or more of the Following Statements may affect this Document 


• This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 


• This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 


• This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 


• This document is paginated as submitted by the original source. 


• Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 


Produced by the NASA Center for Aerospace Information (CASI) 



U>ckheed 

Electronics 

Company, 


A HjnOAIIV IX 
lOCXHIlO UXtKXIAIION 

ISM NASA S<Md I, 770>8 

M. 713 J)J S4II 

Inc. 


JSC-14564 
NUV 1 1978 


Ref: 642-7275 

Contract NAS 9-15200 
Job Order 73.715 


TECHNICAL MEMORANDUM 


ACCURACY ASSESSMENT'S EVALUATION PLAN 


FOR TY MULTILABELING 
By 

A. L. Ona 


"Midi IvilliWi under NASA sponsorsMp 
In ihl Inhrett of early and wide dis- 
ieminili'<n of Earth Resources Survey 
Pninm information and without liability 
IV My «» mode thereof/' 


Approved By: 


E. M. Hsu, Supervisor 
Accuracy Assessment Section 


(E79-10107) accuracy AS"E 
EVALUATICN FIAN FCR TY MU.. 
(Lockheed Flectrorics Co.) 
A01 


SSMENT' S 
rilABELING 

10 p HC A02/MF 

CSCL 05E 


G3/U3 


N79- 17295 


Unclas 

00107 



October 1978 


LEC-12823 


3. Raapwnt'i C«Ulog ^to 


1. Report No. 

JSC-14564 

3 Govarnmant Accauion No 

4 Titl* artd SwMitI* 

ACCURACY ASSESSMENT'S EVALUATION PLAN FOR TY MULTI LABELING 

7 Authorial 

A. L. Oha 

Lockheed Electronics Company, I 

nc. 

9. ^forming Organitation Nam* and Addraa* 

Lockheed Electronics Company, I 
Systems and Services Division 
IfO NASA Road 1 
Houston, Texas 77058 

nc . 


13. Spontorinf Aptncy Nam* top Apdrau 

National Aeronautics and Space Administration 
Lyndon B. Johnson Space Center 
Houston, Texas 77058 




5 Raport Data 

October 1978 


6 RaHorming Organiiation Cod* 


8. Parformmg Organnation Raport No 

LEC-12823 


10 Work Un*t No 


11 Contract or Grant No. 

NAS 9-15200 


13. Typ* of Rapon and Rariod Covarad 

Technical Memorandum 


14 Sporttorinf AgarKy Cod* 



16 APatract 

The plan describes the evaluation of analyst small-grains omission and commission errors 
from Large Area Crop Inventory Experiment Transition Year Multi label Ing In four U.S. 
Northern Great Plains spring wheat states: Montana, North Dakota, Minnesota, and 

South Dakota. The document also describes the coding system used for labeling data and 
the proposed analyses required for the evaluation. Two evaluation plan schedules, for 
Accuracy Assessment ground truth digitization and for manual ground truth Interpretation, 
are presented. 


17. Kay Word! (Suggaitad by AuthorU)) 

I Type 1 dots 
Type 2 dots 

Percent correct labeling 
' Labeling accuracy 
j t-test 

I 


j 19. Security Oa«tf. (of this rtpoM) 

t 

, Unclassified 


30 Sacurity Clawif. (of thit papal 

Unclassified 



*For tala by tti* National Tachnical Information Sarvica. Springfield, Virginia 33161 


JSC ^orm 1424 (R«« Nov 76) 


NASA — JSC 


















1. INTRODUCTION 


Preliminary investigations on labeling error characterization from Phase III 
blind sites in Colorado, Minnesota, Montana, North Dakota, and Oklahoma 
indicate the need for a continued evaluation of omission errors. Transition 
Year (TY) multiple category labeling (called multilabeling) is an extension 
of the current Classification and Mensuration Subsystem (CAMS) processing 
procedures in which the analysts label a variety of crops and categories 
collectively categorized as nonwheat in Phase III. The study will be con- 
ducted to evaluate spring-grains omission and commission errors from TY 
mul ti label ing and to determine if labeling specific crop categories improves 
the labeling accuracy for small grains. To accomplish this task, analyst 
labeling data for the blind sites will be passed to the Accuracy Assessment 
Section in a prescribed computer card format. CAMS will process a total of 
71 blind sites (11 from Minnesota, 15 from Montana, 31 from North Dakota, 
and 14 from South Dakota). It is anticipated that each segment will be 
Interpreted 2 or 3 times during the upcoming spring processing season. 

The purpose of the multilabeling task is to evaluate analyst small-grains 
omission and commission errors in four U.S. Northern Great Plains (USNGP) 
spring wheat states - Montana, North Dakota, Minnesota, and South Dakota. 

The following factors will be studied at the state level: 

a. The confusion crops/categories associated with small-grains omission and 
commission errors and the frequency of occurrence of these errors 

b. The separability of the nonsmal 1 -grains categories and the confusion 
crops associated with these categories 

c. Any significantly recurring crop that is not listed as a nonsmal 1 -grains 
category in table I 

d. If possible, any improvement in the labeling accuracy of small grains 
as a result of the analyst requirements to label specific nonsmall- 
grains categories 
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TABLE I.- mult I LABELING CATEGORIES 


Group I - Small grains, including wheat 


Code 

Description 


Winter small grains 


Spring small grains 


Total winter and spring small grains 

B* 

Barley 


Group II - Field crops, not small grains 


Code 

Description 

mm 

All cultivated hays and grasses, such as alfalfa and clover 


Corn 

mM 

Sorghum 


Sugarbeets 

IS 

Sunflowers 


Soybeans 

mM 

Any identified crop not listed in Group II 


Group III — Other signatures, identified 


Code 

Description 

K 

Idle cropland, clean tilled 

M 

Idle cropland, residue/stubble remaining or weeds/field cover growing 

P 

Natural grasses and pastures 

T 

Trees, timber, and shrubs 

Z 

Nonagriculture, includes lakes, rivers, ponds, sand hills, mountains, 
dry lake beds, highways, cities, airfields, etc. 

X* 

Clouds, haze, shadows, and other obstructions 

Group IV - Other signatures, not identified 

Code 

Description 

N* 

No identification is discernible. 

Note: This code is used o_nJ^ after all other category codes have 

been exhausted. 

*Sent 

to classifier with coded category; all others changed to code N. 
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2. LABELING DATA 


Table I lists the category codes that will be used by CAMS for dot labeling. 
The four groups are: I, small grains; II, nonsmall -grains field crops; III, 

Identified signatures that are neither small-grains nor nonsmall -grains field 

crops; and IV, signatures that are not identified. Labeling type 1 and type 2 

dots for multilabeling remains exactly as established In the CAMS detailed 

analysis procedures except that the analysts substitute a specific nonsmall - 
grains, multilabeling category code !n place of the more general nonwheat 
code ordinarily used. The analyst dot labeling data will be sent to the 
Accuracy Assessment Section on computer cards. 

Type 1 dots are pixels used in the clustering algorithm, which initiates and 
labels clusters. Type 2 dots are pixels used in computation of bias correc- 
tion; this set is completely separate from the dot 1 set. 
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3. TEST APPROACH 


The analyst-interpreter (AI) dot labels for each blind site will be compared 
with the corresponding ground truth (GT) labels and the results will be 
tabulated (fig. 1) at the state level to Identify the confusion crops asso- 
ciated with small -grains omission and commission errors. The aiwlyst dot 
labeling data deck will be processed by the computer routine SPATL to compute 
the frequency of occurrence of the omission and commission errors associated 
with each category. The percent correct labeling for each category will be 
computed at the segment level; for example, for winter small grains. 


W 


(AI • w n GT » w) 
GT total for W 


X 100 


where (AI ■ W n GT * W) Is the number of dots labeled winter small grains 
by both the analysts and the ground truth label. 

Also, the overall labeling accuracy (OLA) for each blind site will be calcu- 
lated as follows: 


OLA » [(AI « W n GT » W) + (AI * S n GT » S) 

+ ... + (AI » N* n GT * N*)]/Grand total 

For each USGP spring wheat state, the average percent correct labeling for 
each category and the average overall labeling accuracy will be determined. 

To assess whether the analyst reguirement to label specific nonsmall -grains 
categories Improves the small -grains labeling accuracy, a t-test^ of the 
average overall labeling accuracy of small-grains and specific nonsmall -grains 
categories (OLAl ) versus the average overall labeling accuracy of small-grains 

"'Test to determine whether the difference between any two treatments Is 
different from zero. 
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and specific nonsma 11 -grains categories (0LA2) will be conducted. OLAl and 
0LA2 will be computed as follows: 

OLAl » [(AI * Grp I n GT * Grp I) + (AI ■ Grp II O 6T ■ Grp II) 

+ (AI » Grp III O GT ■ Grp III)]/Grand total 

OLA 2 - [(AI « Grp I n GT • Gro I) + (AI - H n GT « H) 

+ (AI » C n GT « C) •♦■... + (AI » N* n GT * N*)]/Grand total 

Since each segment will be Interpreted 2 to 3 times during the upcoming spring 
processing season, the changes in labeling accuracy due to additional acqui- 
sitions will also be evaluated. 
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Figure 1 Mul tilabel ing dot distribution by category. 
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4. RESOURCES 


1. Analysts (CAMS) - Nine to 10 CAMS operational analysts will perform TY 
dot mul t1 label Ing during the test period. Estimated 9 to 10 man- 
equivalents (M/E) for 109 man-days (M/D). 

1 . Test Coordinator (Research, Test, and Evaluation Department) - Two M/E 
for approximately 75 M/D to organize, tabulate, and reduce data, run 
Accuracy Assessment (AA) SPATE analysis, document, and present results. 

Two alternatives are given for using the evaluation: AA ground truth digiti- 

zation (Schedule A); and manual ground truth Interpretation (Schedule B). 

Schedule A shows an estimated completion date of mid-March 1979, assuming 
required resources are available when needed. 

Schedule B, an alternative approach for obtaining more timely GT labels, shows 
an estimated completion date In January 1979. This plan uses ground truth 
photographs along with the corresponding field overlays to Identify manually 
the labels of the 209 dots for each blind site. The CAMS dots labels will be 
compared to these corresponding ground truth labels. For each blind site, a 
summary of the dot distribution (refer to fig. 1) by category will be per- 
formed. The percent omission and conmission errors will be computed manually 
instead of using the computer routine SPATE. The results will be tabulated 
and summarized at the state level. 
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